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Abstract 

We describe and analyze the joint source/channel coding properties of a class of sparse 
graphical codes based on compounding a low-density generator matrix (LDGM) code with a 
low-density parity check (LDPC) code. Our first pair of theorems establish that there exist 
codes from this ensemble, with all degrees remaining bounded independently of block length, 
that are simultaneously optimal as both source and channel codes when encoding and decoding 
are performed optimally. More precisely, in the context of lossy compression, we prove that 
finite degree constructions can achieve any pair (i?, D) on the rate-distortion curve of the binary 
symmetric source. In the context of channel coding, we prove that finite degree codes can achieve 
any pair (C, p) on the capacity-noise curve of the binary symmetric channel. Next, we show that 
our compound construction has a nested structure that can be exploited to achieve the Wyner- 
Ziv bound for source coding with side information (SCSI), as well as the Gelfand-Pinsker bound 
T^j- . for channel coding with side information (CCSI). Although the current results are based on 

optimal encoding and decoding, the proposed graphical codes have sparse structure and high 
girth that renders them well-suited to message-passing and other efficient decoding procedures. 
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1 Introduction 

Over the past decade, codes based on graphical constructions, including turbo codes [3] and low- 
density parity check (LDPC) codes [T7], have proven extremely successful for channel coding prob- 
lems. The sparse graphical nature of these codes makes them very well-suited to decoding using 
efficient message-passing algorithms, such as the sum- product and max-product algorithms. The 
asymptotic behavior of iterative decoding on graphs with high girth is well-characterized by the 
density evolution method [39] , which yields a useful design principle for choosing degree dis- 
tributions. Overall, suitably designed LDPC codes yield excellent practical performance under 
iterative message-passing, frequently very close to Shannon limits [7j. 
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However, many other communication problems involve aspects of lossy source coding, either 
alone or in conjunction with channel coding, the latter case corresponding to joint source-channel 
coding problems. Well-known examples include lossy source coding with side information (one 
variant corresponding to the Wyner-Ziv problem [IS]), and channel coding with side information 
(one variant being the Gelfand-Pinsker problem [19] ) . The information-theoretic schemes achieving 
the optimal rates for coding with side information involve delicate combinations of source and 
channel coding. For problems of this nature — in contrast to the case of pure channel coding — the 
use of sparse graphical codes and message-passing algorithm is not nearly as well understood. With 
this perspective in mind, the focus of this paper is the design and analysis sparse graphical codes 
for lossy source coding, as well as joint source/channel coding problems. Our main contribution 
is to exhibit classes of graphical codes, with all degrees remaining bounded independently of the 
blocklength, that simultaneously achieve the information-theoretic bounds for both source and 
channel coding under optimal encoding and decoding. 

1.1 Previous and ongoing work 

A variety of code architectures have been suggested for lossy compression and related problems in 
source/channel coding. One standard approach to lossy compression is via trellis-code quantization 
(TCQ) [26]. The advantage of trellis constructions is that exact encoding and decoding can be 
performed using the max-product or Viterbi algorithm [25j, with complexity that grows linearly 
in the trellis length but exponentially in the constraint length. Various researchers have exploited 
trellis-based codes both for single-source and distributed compression [BJ [231 133 US] as well as 
information embedding problems [3 [T51 I42| . One limitation of trellis-based approaches is the 
fact that saturating rate-distortion bounds requires increasing the trellis constraint length [43J, 
which incurs exponential complexity (even for the max-product or sum-product message-passing 
algorithms) . 

Other researchers have proposed and studied the use of low-density parity check (LDPC) codes 
and turbo codes, which have proven extremely successful for channel coding, in application to 
various types of compression problems. These techniques have proven particularly successful for 
lossless distributed compression, often known as the Slepian-Wolf problem \18\ 140] . An attractive 
feature is that the source encoding step can be transformed to an equivalent noisy channel de- 
coding problem, so that known constructions and iterative algorithms can be leveraged. For lossy 
compression, other work [31j shows that it is possible to approach the binary rate-distortion bound 
using LDPC-like codes, albeit with degrees that grow logarithmically with the blocklength. 

A parallel line of work has studied the use of low-density generator matrix (LDGM) codes, which 
correspond to the duals of LDPC codes, for lossy compression problems [30l 0H O [351 US] • Focusing 
on binary erasure quantization (a special compression problem dual to binary erasure channel 
coding), Martinian and Yedidia [30J proved that LDGM codes combined with modified message- 
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passing can saturate the associated rate-distortion bound. Various researchers have used techniques 
from statistical physics, including the cavity method and replica methods, to provide non-rigorous 
analyses of LDGM performance for lossy compression of binary sources [8, 9\ [35| [M] . In the limit 
of zero-distortion, this analysis has been made rigorous in a sequence of papers [T2J EH CE3]- 
Moreover, our own recent work |28[ [27] provides rigorous upper bounds on the effective rate- 
distortion function of various classes of LDGM codes. In terms of practical algorithms for lossy 
binary compression, researchers have explored variants of the sum-product algorithm [33] or survey 
propagation algorithms [H HI] for quantizing binary sources. 

1.2 Our contributions 

Classical random coding arguments [11] show that random binary linear codes will achieve both 
channel capacity and rate-distortion bounds. The challenge addressed in this paper is the design 
and analysis of codes with bounded graphical complexity, meaning that all degrees in a factor graph 
representation of the code remain bounded independently of blocklength. Such sparsity is critical 
if there is any hope to leverage efficient message-passing algorithms for encoding and decoding. 
With this context, the primary contribution of this paper is the analysis of sparse graphical code 
ensembles in which a low-density generator matrix (LDGM) code is compounded with a low-density 
parity check (LDPC) code (see Fig. [2] for an illustration). Related compound constructions have 
been considered in previous work, but focusing exclusively on channel coding [16[ [36], 141] , In 
contrast, this paper focuses on communication problems in which source coding plays an essential 
role, including lossy compression itself as well as joint source/channel coding problems. Indeed, the 
source coding analysis of the compound construction requires techniques fundamentally different 
from those used in channel coding analysis. We also note that the compound code illustrated 
in Fig. [5] can be applied to more general memoryless channels and sources; however, so as to bring 
the primary contribution into sharp focus, this paper focuses exclusively on binary sources and/or 
binary symmetric channels. 

More specifically, our first pair of theorems establish that for any rate R € (0, 1), there exist 
codes from compound LDGM/LDPC ensembles with all degrees remaining bounded independently 
of the blocklength that achieve both the channel capacity and the rate-distortion bound. To the 
best of our knowledge, this is the first demonstration of code families with bounded graphical 
complexity that are simultaneously optimal for both source and channel coding. Building on 
these results, we demonstrate that codes from our ensemble have a naturally "nested" structure, 
in which good channel codes can be partitioned into a collection of good source codes, and vice 
versa. By exploiting this nested structure, we prove that codes from our ensembles can achieve 
the information-theoretic limits for the binary versions of both the problem of lossy source coding 
with side information (SCSI, known as the Wyner-Ziv problem [IS]), and channel coding with side 
information (CCSI, known as the Gelfand-Pinsker [19] problem). Although these results are based 
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on optimal encoding and decoding, a code drawn randomly from our ensembles will, with high 
probability, have high girth and good expansion, and hence be well-suited to message-passing and 
other efficient decoding procedures. 

The remainder of this paper is organized as follows. Section [2] contains basic background 
material and definitions for source and channel coding, and factor graph representations of binary 
linear codes. In Section El we define the ensembles of compound codes that are the primary 
focus of this paper, and state (without proof) our main results on their source and channel coding 
optimality. In Section [H we leverage these results to show that our compound codes can achieve 
the information-theoretic limits for lossy source coding with side information (SCSI), and channel 
coding with side information (CCSI). Sections [5] and [6] are devoted to proofs that codes from the 
compound ensemble are optimal for lossy source coding (Section [5]) and channel coding (Section [6]) 
respectively. We conclude the paper with a discussion in Section [7J Portions of this work have 
previously appeared as conference papers [281 [291 [27] . 

2 Background 

In this section, we provide relevant background material on source and channel coding, binary 
linear codes, as well as factor graph representations of such codes. 

2.1 Source and channel coding 

A binary linear code C of block length n consists of all binary strings x € {0, l} n satisfying a 
set of m < n equations in modulo two arithmetic. More precisely, given a parity check matrix 
H E {0, l} mxn ; the code is given by the null space 

C := {xG{0,l} n | Hx = 0}. (1) 

Assuming the parity check matrix H is full rank, the code C consists of 2 n_m = 2 nR codewords, 
where R = 1 — ^ is the code rate. 

Channel coding: In the channel coding problem, the transmitter chooses some codeword x € C 
and transmits it over a noisy channel, so that the receiver observes a noise-corrupted version Y. 
The channel behavior is modeled by a conditional distribution F(y \ x) that specifies, for each 
transmitted sequence Y, a probability distribution over possible received sequences {Y = y}. In 
many cases, the channel is memoryless, meaning that it acts on each bit of C in an independent 
manner, so that the channel model decomposes as F(y \ x) = JlILi fi{ x H Hi) Here each function 
fi(xi',yi) = P(yi | Xi) is simply the conditional probability of observing bit yi given that Xi was 
transmitted. As a simple example, in the binary symmetric channel (BSC), the channel flips each 
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transmitted bit Xi with probability p, so that V(yi \ Xi) = (1 — p)I[xi = yi\ + p(l — I[x?, ^ yi]), 
where 1(A) represents an indicator function of the event A. With this set-up, the goal of the re- 
ceiver is to solve the channel decoding problem: estimate the most likely transmitted codeword, 
given by x : = argmaxP(y | x). The Shannon capacity [TT] of a channel specifies an upper bound 
on the rate R of any code for which transmission can be asymptotically error-free. Continuing 
with our example of the BSC with flip probability p, the capacity is given by C = 1 — h(p), where 
h(p) : = — p log 2 p — (1 — p) log 2 (l — p) is the binary entropy function. 

Lossy source coding: In a lossy source coding problem, the encoder observes some source se- 
quence S £ S, corresponding to a realization of some random vector with i.i.d. elements Si ~ P5. 
The idea is to compress the source by representing each source sequence S by some codeword 
x £ C. As a particular example, one might be interested in compressing a symmetric Bernoulli 
source, consisting of binary strings S S {0, l} n , with each element S{ drawn in an independent 
and identically distributed (i.i.d.) manner from a Bernoulli distribution with parameter p = ^. 
One could achieve a given compression rate R = ^ by mapping each source sequence to some 
codeword x € C from a code containing 2 m = 2 nR elements, say indexed by the binary sequences 
z £ {0, l} m . In order to assess the quality of the compression, we define a source decoding map 
x 1 — * S(x), which associates a source reconstruction S(x) with each codeword x £ C. Given some 
distortion metric d : S x S — > R+, the source encoding problem is to find the codeword with min- 
imal distortion — namely, the optimal encoding x := axgmind(S(x),S). Classical rate-distortion 
theory [11] specifies the optimal trade-offs between the compression rate R and the best achievable 
average distortion D = M[d(S, S)], where the expectation is taken over the random source sequences 
S. For instance, to follow up on the Bernoulli compression example, if we use the Hamming metric 
d(S,S) = i Y27=i \Si ~ Si\ as the distortion measure, then the rate-distortion function takes the 
form R(D) = 1 — h(D), where h is the previously defined binary entropy function. 

We now provide definitions of "good" source and channel codes that are useful for future reference. 

Definition 1. (a) A code family is a good D-distortion binary symmetric source code if for any 
e > 0, there exists a code with rate R < 1 — h (D) + e that achieves Hamming distortion less than 
or equal to D. 

(b) A code family is a good BSC(p)-noise channel code if for any e > there exists a code with 
rate R > 1 — h (p) — e with error probability less than e. 

2.2 Factor graphs and graphical codes 

Both the channel decoding and source encoding problems, if viewed naively, require searching over 
an exponentially large codebook (since |C| = 2 nR for a code of rate R). Therefore, any practically 
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useful code must have special structure that facilitates decoding and encoding operations. The 
success of a large subclass of modern codes in use today, especially low-density parity check (LDPC) 
codes [UlEH], ^ s based on the sparsity of their associated factor graphs. 
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(a) (b) 

Figure 1. (a) Factor graph representation of a rate R = 0.5 low-density parity check (LDPC) code 
with bit degree d v = 3 and check degree d' c — 6. (b) Factor graph representation of a rate R = 0.75 
low-density generator matrix (LDGM) code with check degree d c = 3 and bit degree d v = 4. 



Given a binary linear code C, specified by parity check matrix H, the code structure can be 
captured by a bipartite graph, in which circular nodes (O) represent the binary values Xj (or columns 
of H), and square nodes (■) represent the parity checks (or rows of H). For instance, Fig. QJa) 
shows the factor graph for a rate R = ^ code in parity check form, with m = 6 checks acting on 
n = 12 bits. The edges in this graph correspond to l's in the parity check matrix, and reveal the 
subset of bits on which each parity check acts. The parity check code in Fig. Ufa) is a regular 
code with bit degree 3 and check degree 6. Such low-density constructions, meaning that both the 
bit degrees and check degrees remain bounded independently of the block length n, are of most 
practical use, since they can be efficiently represented and stored, and yield excellent performance 
under message-passing decoding. In the context of a channel coding problem, the shaded circular 
nodes at the top of the low-density parity check (LDPC) code in panel (a) represent the observed 
variables yi received from the noisy channel. 

Figure [D(b) shows a binary linear code represented in factor graph form by its generator matrix 
G. In this dual representation, each codeword x G {0, l} n is generated by taking the matrix-vector 
product of the form Gz, where z € {0, l} m is a sequence of information bits, and G € {0, l} nxm is 
the generator matrix. For the code shown in panel (b), the blocklength is n = 12, and information 
sequences are of length m = 9, for an overall rate of R = m/n = 0.75 in this case. The degrees of 
the check and variable nodes in the factor graph are d c = 3 and d v = 4 respectively, so that the 
associated generator matrix G has d c = 3 ones in each row, and d v = 4 ones in each column. When 
the generator matrix is sparse in this setting, then the resulting code is known as a low- density 
generator matrix (LDGM) code. 
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2.3 Weight enumerating functions 

For future reference, it is useful to define the weight enumerating function of a code. Given a binary 
linear code of blocklength m, its codewords x have renormalized Hamming weights w : = that 
range in the interval [0, 1]. Accordingly, it is convenient to define a function W m : [0, 1] — » R+ that, 
for each w G [0, 1], counts the number of codewords of weight w. 



x G C 



w 



~\\ x \W 


} 


m 





(2) 



where [■] denotes the ceiling function. Although it is typically difficult to compute the weight enu- 
merator itself, it is frequently possible to compute (or bound) the average weight enumerator, where 
the expectation is taken over some random ensemble of codes. In particular, our analysis in the 
sequel makes use of the average weight enumerator of a (d v , d' c )-regular LDPC code (see Fig. H^a)), 
defined as 



A m (w;d v ,d' c ) 



m 



■logE[W m H] 



(3) 



where the expectation is taken over the ensemble of all regular (d v , cQ-LDPC codes. For such 
regular LDPC codes, this average weight enumerator has been extensively studied in previous 
work PUE2]. 



3 Optimality of bounded degree compound constructions 

In this section, we describe the compound LDGM/LDPC construction that is the focus of this 
paper, and describe our main results on their source and channel coding optimality. 

3.1 Compound construction 

Our main focus is the construction illustrated in Fig. [2J obtained by compounding an LDGM code 
(top two layers) with an LDPC code (bottom two layers). The code is defined by a factor graph 
with three layers: at the top, a vector x G {0, l} n of codeword bits is connected to a set of n parity 
checks, which are in turn connected by a sparse generator matrix G to a vector y G {0, l} m of 
information bits in the middle layer. The information bits y are also codewords in an LDPC code, 
defined by the parity check matrix H connecting the middle and bottom layers. 

In more detail, considering first the LDGM component of the compound code, each codeword 
x E {0, l} n in the top layer is connected via the generator matrix G G {0, l} nxm to an information 
sequence y G {0, l} m in the middle layer; more specifically, we have the algebraic relation x = Gy. 
Note that this LDGM code has rate Rg < Second, turning to the LDPC component of the 
compound construction, its codewords correspond to a subset of information sequences y G {0, l} m 



7 



in the middle layer. In particular, any valid codeword y satisfies the parity check relation Hy = 0, 
where H G {0, l} mxfc joins the middle and bottom layers of the construction. Overall, this defines 
an LDPC code with rate Rh = 1 — assuming that H has full row rank. 

The overall code C obtained by concatenating the LDGM and LDPC codes has blocklength n, 
and rate R upper bounded by RqRh- In algebraic terms, the code C is defined as 

C : = {x G {0, l} n | x = Gy for some y G {0, l} m such that Hy = 0} , (4) 

where all operations are in modulo two arithmetic. 




ki k 2 



Figure 2. The compound LDGM/LDPC construction analyzed in this paper, consisting of a (n, m) 
LDGM code over the middle and top layers, compounded with a (m, k) LDPC code over the middle 
and bottom layers. Codewords x € {0, 1}" are placed on the top row of the construction, and are 
associated with information bit sequences z G {0, l} m in the middle layer. The LDGM code over the 
top and middle layers is defined by a sparse generator matrix G G {0, 1}™ X ™ 1 with at most d c ones 
per row. The bottom LDPC over the middle and bottom layers is represented by a sparse parity 
check matrix H G {0, lj kxm with d v ones per column, and d' c ones per row. 

Our analysis in this paper will be performed over random ensembles of compound LDGM/LDPC 
ensembles. In particular, for each degree triplet (d c ,d v ,d' c ), we focus on the following random 
ensemble: 

(a) For each fixed integer d c > 4, the random generator matrix G G {0, l} nxm is specified as 
follows: for each of the n rows, we choose d c positions with replacement, and put a 1 in each 
of these positions. This procedure yields a random matrix with at most d c ones per row, 
since it is possible (although of asymptotically negligible probability for any fixed d c ) that 
the same position is chosen more than once. 

(b) For each fixed degree pair (d v ,d' c ), the random LDPC matrix H G {0, l} fcxm is chosen uni- 
formly at random from the space of all matrices with exactly d v ones per column, and exactly 
d' c ones per row. This ensemble is a standard (d v , d(,)-regular LDPC ensemble. 
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We note that our reason for choosing the check-regular LDGM ensemble specified in step (a) is not 
that it need define a particularly good code, but rather that it is convenient for theoretical purposes. 
Interestingly, our analysis shows that the bounded degree d c check-regular LDGM ensemble, even 
though it is sub-optimal for both source and channel coding in isolation [2SJJ22], defines optimal 
source and channel codes when combined with a bottom LDPC code. 

3.2 Main results 

Our first main result is on the achievability of the Shannon rate-distortion bound using codes from 
LDGM/LDPC compound construction with finite degrees (d c ,d v ,d' c ). In particular, we make the 
following claim: 

Theorem 1. Given any pair (R,D) satisfying the Shannon bound, there is a set of finite degrees 
(d c ,d v ,d' c ) and a code from the associated LDGM/LDPC ensemble with rate R that is a D-good 
source code (see Definition^). 

In other work |28[ I27|. we showed that standard LDGM codes from the check-regular ensemble 
cannot achieve the rate-distortion bound with finite degrees. As will be highlighted by the proof of 
Theorem [T] in Section [SJ the inclusion of the LDPC lower code in the compound construction plays 
a vital role in the achievability of the Shannon rate-distortion curve. 

Our second main result of this result is complementary in nature to Theorem [H regarding the 
achievability of the Shannon channel capacity using codes from LDGM/LDPC compound construc- 
tion with finite degrees (d c ,d v ,d' c ). In particular, we have: 

Theorem 2. For all rate-noise pairs (R,p) satisfying the Shannon channel coding bound R < 
1 — h(p), there is a set of finite degrees (d c ,d v ,d' c ) and a code from the associated LDGM/LDPC 
ensemble with rate R that is a p-good channel code (see Definition^). 

To put this result into perspective, recall that the overall rate of this compound construction is 
given by R = RgRh- Note that an LDGM code on its own (i.e., without the lower LDPC code) 
is a special case of this construction with Rh = 1. However, a standard LDGM of this variety is 
not a good channel code, due to the large number of low-weight codewords. Essentially, the proof 
of Theorem [2] (see Section [6]) shows that using a non-trivial LDPC lower code (with Rh < 1) can 
eliminate these troublesome low- weight codewords. 

4 Consequences for coding with side information 

We now turn to consideration of the consequences of our two main results for problems of coding 
with side information. It is well-known from previous work [UJ that achieving the information- 
theoretic limits for these problems requires nested constructions, in which a collection of good source 
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codes are nested inside a good channel code (or vice versa). Accordingly, we begin in Section [4,11 
by describing how our compound construction naturally generates such nested ensembles. In Sec- 
tions 14.21 and 14.31 respectively, we discuss how the compound construction can be used to achieve 
the information-theoretic optimum for binary source coding with side information (a version of the 
Wyner-Ziv problem |45j), and binary information embedding (a version of "dirty paper coding", 
or the Gelfand-Pinsker problem |19j). 

4.1 Nested code structure 

The structure of the compound LDGM/LDPC construction lends itself naturally to nested code 
constructions. In particular, we first partition the set of k lower parity checks into two disjoint 
subsets K\ and K2, of sizes k\ and respectively, as illustrated in Fig. [2j Let H\ and H2 denote 
the corresponding partitions of the full parity check matrix H G {0, l} fcxm . Now let us set all parity 
bits in the subset K2 equal to zero, and consider the LDGM/LDPC code C(G,H\) defined by the 
generator matrix G and the parity check (sub)matrix Hi, as follows 

C(G, Hi) := {x G {0, l} n I x = Gy for some y G {0, l} m such that Hi y = 0} . (5) 

Note that the rate of C(G, is given by R' = Rq Rh^ which can be suitably adjusted by 
modifying the LDGM and LDPC rates respectively. Moreover, by applying Theorems Q] and [21 
there exist finite choices of degree such that C(G,H{) will be optimal for both source and channel 
coding. 

Considering now the remaining hi parity bits in the subset K2, suppose that we set them equal 
to a fixed binary sequence m G {0, l} fc2 . Now consider the code 

C(m) : = jx G {0, l} n | x = Gy for some y G {0, l} m such that 

Note that for each binary sequence m G {0, l} ki , the code C(m) is a subcode of C(G, Hi); moreover, 
the collection of these subcodes forms a disjoint partition as follows 

C(G,Hi) = |J C(m). (7) 

me{0,l}'2 

Again, Theorems Q] and [5] guarantee that (with suitable degree choices), each of the subcodes C(m) 
is optimal for both source and channel coding. Thus, the LDGM/LDPC construction has a natural 
nested property, in which a good source/channel code — namely C(G,Hi) — is partitioned into a 
disjoint collection {C(m), m G {0, l} fcl } of good source/channel codes. We now illustrate how this 
nested structure can be exploited for coding with side information. 
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4.2 Source coding with side information 

We begin by showing that the compound construction can be used to perform source coding with 
side information (SCSI). 



4.2.1 Problem formulation 

Suppose that we wish to compress a symmetric Bernoulli source S ~ Ber(^) so as to be able 
to reconstruct it with Hamming distortion D. As discussed earlier in Section [21 the minimum 
achievable rate is given by R{D) = 1 — h (D) . In the Wyner-Ziv extension of standard lossy com- 
pression [35|, there is an additional source of side information about S — say in the form Z = S W 
where W ~ Ber((5) is observation noise — that is available only at the decoder. See Fig. [3] for a block 
diagram representation of this problem. 





R 




] - 





w 



Decoder 



z 



Figure 3. Block diagram representation of source coding with side information (SCSI). A source S is 
compressed to rate R. The decoder is given the compressed version, and side information Z = S®W, 
and wishes to use (S, Z) to reconstruct the source S up to distortion D. 



For this binary version of source coding with side information (SCSI), it is known [2] that the 
minimum achievable rate takes the form 

Rwz(D,p)=l.c.e.{h(D*p)-h{D), (p,0)}, (8) 

where 1. c. e. denotes the lower convex envelope. Note that in the special case p = |, the side 
information is useless, so that the Wyner-Ziv rate reduces to classical rate-distortion. In the 
discussion to follow, we focus only on achieving rates of the form h (D *p) — h (D), as any remaining 
rates on the Wyner-Ziv curve ([8]) can be achieved by time-sharing with the point (p, 0). 

4.2.2 Coding procedure for SCSI 

In order to achieve rates of the form R = h(D * p) — h (D), we use the compound LDGM/LDPC 
construction, as illustrated in Fig. [2 according to the following procedure. 

Step #1, Source coding: The first step is a source coding operation, in which we transform the 
source sequence S to a quantized representation S. In order to do so, we use the code C(G, Hi), as 
defined in equation ([5]) and illustrated in Fig. 0(a), composed of the generator matrix G and the 
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parity check matrix H x . Note that C(G, H\), when viewed as a code with blocklength n, has rate 

fi— — 1 n 

^1 := — n m = m ~n 1 ■ Suppose that we choosqll the middle and lower layer sizes m and ki 
respectively such that 

Ri = = l-h(D)+e/2, (9) 

where e > is arbitrary. For any such choice, Theorem [1] guarantees the existence of finite degrees 
(d c , d v , d' c ) such that that C(G, Hi) is a good D-distortion source code. Consequently, for the speci- 
fied rate Ri, we can use C(G, Hi) in order to transform the source to some quantized representation 
S such that the error S © S has average Hamming weighted bounded by D. Moreover, since S is a 
codeword of C(G, H\), there is some sequence of information bits Y £ {0, l} m such that S = GY 
and H X Y = 0. 




ki k\ k 2 

(a) (b) 

Figure 4. (a) Source coding stage for Wyner-Ziv procedure: the C(G, Hi), specified by the generator 
matrix G G {0, l}" xm and parity check matrix Hi G {0, l} fe i Xm ) is used to quantize the source vector 
S G {0, 1}™, thereby obtaining a quantized version S G {0, 1}™ and associated vector of information 
bits Y G {0, l} m , such that S = GY and H x Y = 0. 



Step #2. Channel coding: Given the output (Y, S) of the source coding step, consider the 
sequence H^Y G {0, l} k ' 2 of parity bits associated with the parity check matrix H 2 . Transmitting 
this string of parity bits requires rate -Rtrans = „ ■ Overall, the decoder receives both these k 2 parity 
bits, as well as the side information sequence Z = S © W. Using these two pieces of information, 
the goal of the decoder is to recover the quantized sequence S. 

Viewing this problem as one of channel coding, the effective rate of this channel code is 
R2 = m ~ J,~ ■ Note that the side information can be written in the form 

Z = <S©TF = £©£ffiVF, 

Note that the choices of m and fci need not be unique. 
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where E : = S(BS is the quantization noise, and W ~ Ber(p) is the channel noise. If the quantization 
noise E were i.i.d. Ber(D), then the overall effective noise E © W would be i.i.d. Ber(D *p). (In 
reality, the quantization noise is not exactly i.i.d. Ber(D), but it can be shown [17] that it can be 
treated as such for theoretical purposes.) Consequently if we choose ki such that 

„ m — k-i — ko / \ i , 

R 2 = = l-h(D*p)-e 2, (10) 

n 

for an arbitrary e > 0, then Theorem [5] guarantees that the decoder will (w.h.p.) be able to recover 
a codeword corrupted by (D * p)-Bernoulli noise. 



Summarizing our findings, we state the following: 

Corollary 1. There exist finite choices of degrees (d c , d v , d' c ) such that the compound LDGM/LDPC 
construction achieves the Wyner-Ziv bound. 

Proof. With the source coding rate Ri chosen according to equation ([9]), the encoder will return a 
quantization S with average Hamming distance to the source S of at most D. With the channel 
coding rate R2 chosen according to equation (fTO]) . the decoder can with high probability recover 
the quantization S. The overall transmission rate of the scheme is 

R h 

-fi-trans — 

n 

m — k% m — k\ — h% 
n n 
= Ri — R2 

= (l-h(D) + e/2)-(l-h(D*p)-e/2) 
= h{D*p)-h{D) + e. 

Since e > was arbitrary, we have established that the scheme can achieve rates arbitrarily close 
to the Wyner-Ziv bound. □ 

4.3 Channel coding with side information 

We now show how the compound construction can be used to perform channel coding with side 
information (CCSI). 
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4.3.1 Problem formulation 



In the binary information embedding problem, given a specified input vector V £ {0, l} n , the 
channel output Z £ {0, l} n is assumed to take the form 



v®s®w, 



(11) 



where S is a host signal (not under control of the user), and W ~ Ber(p) corresponds to channel 
noise. The encoder is free to choose the input vector V £ {0, l} n , subject to an average channel 
constraint 



■n 



®[\\V\\i] 



< 



UK 



(12) 



for some parameter w € (0, 5]. The goal is to use a channel coding scheme that satisfies this 
constraint (|12h so as to maximize the number of possible messages m that can be reliably commu- 
nicated. Moreover, We write V = V m to indicate that each channel input is implicitly identified 
with some underlying message m. Given the channel output Z = V m © S © W, the goal of the de- 
coder is to recover the embedded message m. The capacity for this binary information embedding 
problem [2] is given by 



Ri E (w,p) 



u. c. e 



{h(w)-h(p), (0,0)}, 



(13) 



where u. c. e. denotes the upper convex envelope. As before, we focus on achieving rates of the form 
h (w) — h(p), since any remaining points on the curve (|13H can be achieved via time-sharing with 
the (0, 0) point. 



W 



m 



-f Encoder 



Vrr 



{} 



v Z 

£3 f Decoder V 



m 



S 

Figure 5. Block diagram representation of channel coding with side information (CCSI). The 
encoder embeds a message m into the channel input V m , which is required to satisfy the average 
channel constraint ~E[||Vin||i] < w. The channel produces the output Z = V m © S ® W, where S is a 
host signal known only to the encoder, and W ~ Ber(p) is channel noise. Given the channel output 
Y, the decoder outputs an estimate m of the embedded message. 
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4.3.2 Coding procedure for CCSI 

In order to achieve rates of the form R = h (w) — h(p), we again use the compound LDGM/LDPC 
construction in Fig. [21 now according to the following two step procedure. 

Step #1: Source coding: The goal of the first stage is to embed the message into the transmitted 
signal V via a quantization process. In order to do so, we use the code illustrated in Fig. [6ja), 
specified by the generator matrix G and parity check matrices H\ and Hi- The set K\ of k\ parity 
bits associated with the check matrix H\ remain fixed to zero throughout the scheme. On the other 
hand, we use the remaining k 2 lower parity bits associated with H 2 to specify a particular message 
m G {0, l} k2 that the decoder would like to recover. In algebraic terms, the resulting code C(m) 
has the form 



C(m) : = { x G {0, l} n | x = Gy for some y G {0, l} m such that 










1 


y = 






H 2 _ 


m 





.(14) 



Since the encoder has access to host signal S, it may use this code C(m) in order to quantize the 
host signal. After doing so, the encoder has a quantized signal S m G {0, l} n and an associated 
sequence Y m G {0, l} m of information bits such that S m = GY m . Note that the quantized signal 
(Y m ,S m ) specifies the message m in an implicit manner, since m = H%Y m by construction of the 
code C(m). 

Now suppose that we choose n, m and k such that 

Ft! = m ~ fcl ~ fc2 = l-h(w) + e/2 (15) 
n 

for some e > 0, then Theorem [1] guarantees that there exist finite degrees (d c ,d v ,d' c ) such that 
the resulting code is a good ^-distortion source code. Otherwise stated, we are guaranteed that 
w.h.p, the quantization error E := S S has average Hamming weight upper bounded by uun. 
Consequently, we may set the channel input V equal to the quantization noise (V = E), thereby 
ensuring that the average channel constraint (|12|) is satisfied. 



Step #2, Channel coding: In the second phase, the decoder is given a noisy channel observation 
of the form 

Z = E@S@W = S®W, (16) 

and its task is to recover S. In terms of the code architecture, the k\ lower parity bits remain set to 
zero; the remaining k 2 parity bits, which represent the message m, are unknown to the coder. The 
resulting code, as illustrated illustrated in Fig. [U(b), can be viewed as channel code with effective 
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m 

(a) (b) 

Figure 6. (a) Source coding step for binary information embedding. The message m 6 {0, l} fe2 
specifies a particular coset; using this particular source code, the host signal S is compressed to S, 
and the quantization error E — S © S is transmitted over the constrained channel, (b) Channel 
coding step for binary information embedding. The decoder receives Z — S © W where W ~ Ber(p) 
is channel noise, and seeks to recover S, and hence the embedded message m specifying the coset. 



rate Now suppose that we choose k\ such that the effective code used by the decoder has 

rate 

R 2 = I^LZ*I = i_fc(p)_ e /2, (17) 
n 

for some e > 0. Since the channel noise W is Ber(p) and the rate R2 chosen according to (|17|) . 
Theorem [2] guarantees that the decoder will w.h.p. be able to recover the pair S and Y. Moreover, 
by design of the quantization procedure, we have the equivalence m = H2 Y so that a simple 
syndrome-forming procedure allows the decoder to recover the hidden message. 
Summarizing our findings, we state the following: 

Corollary 2. There exist finite choices of degrees (d c , d v , d' c ) such that the compound LDGM/LDPC 
construction achieves the binary information embedding ( Gelfand-Pinsker) bound. 

Proof. With the source coding rate Ri chosen according to equation (|15p . the encoder will return 
a quantization 5* of the host signal S with average Hamming distortion upper bounded by w. 
Consequently, transmitting the quantization error E = S © S will satisfy the average channel 
constraint (fT2|) . With the channel coding rate R2 chosen according to equation (fTT|) . the decoder 
can with high probability recover the quantized signal S, and hence the message m. Overall, the 
scheme allows a total of 2 k2 distinct messages to be embedded, so that the effective information 
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embedding rate is 



-'•-trans — 

n 

m — k\ m — k\ — k<i 



n n 
= R2 — Ri 

= (1 - /» (p) - e/2) - (1 - /» (w) + e/2) 
= h(w) - h (p) + e, 

for some e > 0. Thus, we have shown that the proposed scheme achieves the binary information 
embedding bound (fT3|) . □ 



5 Proof of source coding optimality 

This section is devoted to the proof of the previously stated Theorem [T] on the source coding 
optimality of the compound construction. 

5.1 Set-up 

In establishing a rate-distortion result such as Theorem [H perhaps the most natural focus is the 
random variable 

d n (S,C) := — min llx — 5|h, (18) 

n zee 

corresponding to the (renormalized) minimum Hamming distance from a random source sequence 
S £ {0, l} n to the nearest codeword in the code C. Rather than analyzing this random variable 
directly, our proof of Theorem [1] proceeds indirectly, by studying an alternative random variable. 

Given a binary linear code with N codewords, let i = 0, 1, 2, . . . , N — 1 be indices for the 
different codewords. We say that a codeword X 1 is distortion D-good for a source sequence S if the 
Hamming distance \\X l ®S\\\ is at most Dn. We then set the indicator random variable Z l {D) = 1 
when codeword X 1 is distortion D-good. With these definitions, our proof is based on the following 
random variable: 

JV-l 

T n (S,C;D) := (19) 

Note that T n (S,C;D) simply counts the number of codewords that are distortion D-good for a 
source sequence S. Moreover, for all distortions D, the random variable T n (S, C;D) is linked to 
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d n (S,C) via the equivalence 

F[T n (S,C;D) > 0] = P[d n (5,C) <D}. (20) 

Throughout our analysis of ¥[T n (S, C; D) > 0], we carefully track only its exponential behavior. 
More precisely, the analysis to follow will establish an inverse polynomial lower bound of the 
form F[T n (S, C; D) > 0] > l//(n) where /(•) collects various polynomial factors. The following 
concentration result establishes that the polynomial factors in these bounds can be ignored: 

Lemma 1 (Sharp concentration). Suppose that for some target distortion D, we have 

F[T n (S,C;D) >0] > l//(n), (21) 

where /(•) is a polynomial function satisfying log/(n) = o(n). Then for all e > 0, there exists a 
fixed code C of sufficiently large blocklength n such that E[d n (S';C)] < D + e. 

Proof. Let us denote the random code C as (Ci,C2), where Ci denotes the random LDGM top 
code, and C2 denotes the random LDPC bottom code. Throughout the analysis, we condition on 
some fixed LDPC bottom code, say C2 = C-2- We begin by showing that the random variable 
(d n (S, C) I C2) is sharply concentrated. In order to do so, we construct a vertex-exposure martin- 
gale [33j of the following form. Consider a fixed sequential labelling {1, . . . ,n} of the top LDGM 
checks, with check i associated with source bit S{. We reveal the check and associated source 
bit in a sequential manner for each i = 1, ... ,n, and so define a sequence of random variables 
{U , U x ,...,U n } via U Q : = E[d n {S, C) | C 2 ], and 

Ui := E [d n (S,C) I Si,.. .,Si, C 2 ] , i = l,...,n. (22) 

By construction, we have U n = (d n (S,C) \ C2). Moreover, this sequence satisfies the following 
bounded difference property: adding any source bit Si and the associated check in moving from 
Ui-i to U can lead to a (renormalized) change in the minimum distortion of at most Cj = 1/n. 
Consequently, by applying Azuma's inequality [TJ, we have, for any e > 0, 

P [| (d n (S, C) I C 2 ) - E[d n (S, C) I C 2 ]| > e] < exp (-ne 2 ) . (23) 

Next we observe that our assumption (|2ip of inverse polynomial decay implies that, for at least 
one bottom code C2, 

F[d n (S,C)<D I C 2 ] = F[T n (S,C;D) >0 | C 2 ] > l/g{n), (24) 
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for some subexponential function g. Otherwise, there would exist some a > such that 

¥[T n (S,C;D) > | C 2 ] < exp(-na) 

for all choices of bottom code C 2 , and taking averages would violate our assumption (|2ip. 

Finally, we claim that the concentration result (I23j) and inverse polynomial bound (124j) yield 
the result. Indeed, if for some e > 0, we had D < K[d n (S,C) | C 2 ] — e, then the concentration 
bound (|23p would imply that the probability 

nd n (S,C) < D I C a ] < F[d n (S,C) < Hd n (S,C) I C 2 ] - e | C 2 ] 

< P[|(d n (S,C) I C 2 )-E[d n (S 5 C) |C 2 ]| >e] 

decays exponentially, which would contradict the inverse polynomial bound (|24p for sufficiently 
large n. Thus, we have shown that assumption (|2ip implies that for all e > 0, there exists a 
sufficiently large n and fixed bottom code C 2 such that K[d n (S, C) | C 2 ] < D + e. If the average 
over LDGM codes Ci satisfies this bound, then at least one choice of LDGM top code must also 
satisfy it, whence we have established that there exists a fixed code C such that K[d n (S; C)] < D+e, 
as claimed. □ 



5.2 Moment analysis 

In order to analyze the probability ¥[T n (S,C;D) > 0], we make use of the moment bounds given 
in the following elementary lemma: 

Lemma 2 (Moment methods). Given any random variable N taking non-negative integer values, 
there holds 

(E[AH) 2 (a) (6) 

< P[iV>0] < E[N}. (25) 

Proof. The upper bound (b) is an immediate consequence of Markov's inequality, whereas the lower 
bound (a) follows by applying the Cauchy-Schwarz inequality [20] as follows 

(E[7V]) 2 = (E[iVI[iV > 0]]) 2 < EfA^ 2 ] E [l 2 [iV > 0]] = E[iV 2 ] F[N > 0]. 

□ 

The remainder of the proof consists in applying these moment bounds to the random variable 
T n (S, C; D), in order to bound the probability P[T n (5, C; D) > 0]. We begin by computing the first 
moment: 
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Lemma 3 (First moment). For any code with rate R, the expected number of D- good codewords 
scales exponentially as 

-logE[T n ] = [R - {l-h (£>))] ± o(l). (26) 
n 

Proof. First, by linearity of expectation E[T n ] = ^"o" 1 ^[^(-D) = 1] = 2 nR P[Z°( J D) = 1], where 
we have used symmetry of the code construction to assert that ¥[Z l (D) = 1] = F[Z°(D) = 1] for 
all indices i. Now the event {Z°(D) = 1} is equivalent to an i.i.d Bernoulli(^) sequence of length 
n having Hamming weight less than or equal to Dn. By standard large deviations theory (either 
Sanov's theorem or direct asymptotics of binomial coefficients), we have 

-log¥[Z°(D) = 1] = l-h(D)±o(l), 
n 

which establishes the claim. 

□ 

Unfortunately, however, the first moment E[T n ] need not be representative of typical behavior 
of the random variable T n , and hence overall distortion performance of the code. As a simple 
illustration, consider an imaginary code consisting of 2 nR copies of the all-zeroes codeword. Even 
for this "code", as long as R > 1 — h (D), the expected number of distortion- Z) optimal codewords 
grows exponentially. Indeed, although T n = for almost all source sequences, for a small subset of 
source sequences (of probability mass ~ 2~ n I 1 ^^- )] ), the random variable T n takes on the enormous 
value 2 nR , so that the first moment grows exponentially. However, the average distortion incurred 
by using this code will be ~ 0.5 for any rate, so that the first moment is entirely misleading. In order 
to assess the representativeness of the first moment, one needs to ensure that it is of essentially the 
same order as the variance, hence the comparison involved in the second moment bound (|25|) (a) . 

5.3 Second moment analysis 

Our analysis of the second moment begins with the following alternative representation: 
Lemma 4. 

E[T n 2 (D)] = E[T n (D)] (l + {j2nZ j (D) = 1 | Z°(D) = 1]} ). (27) 

Based on this lemma, proved in Appendix [Cl we see that the key quantity to control is the condi- 
tional probability P[Z 3 (D) = 1 | Z (D) = 1]. It is this overlap probability that differentiates the 
low-density codes of interest here from the unstructured codebooks used in classical random coding 
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arguments^] For a low-density graphical code, the dependence between the events {Z^(D) = 1} 
and {Z (D) = 1} requires some analysis. 

Before proceeding with this analysis, we require some definitions. Recall our earlier definition ([3]) 
of the average weight enumerator associated with an (g?^,^) LDPC code, denoted by A. m (tt?). 
Moreover, let us define for each w € [0, 1] the probability 

Q(w;D) := P [\\X(w) © S\\t < Dn | < Dn] , (28) 

where the quantity X(w) € {0, l} n denotes a randomly chosen codeword, conditioned on its under- 
lying length-m information sequence having Hamming weight \wrn~] . As shown in Lemma [9] (see 
Appendix [A}, the random codeword X(w) has i.i.d. Bernoulli elements with parameter 



5*(w;d c ) = - 



1- (1- 2w) 



(29) 



With these definitions, we now break the sum on the RHS of equation (|27ft into m terms, indexed 
hyt = 1,2,... ,m, where term t represents the contribution of a given non-zero information sequence 
y € {0, l} m with (Hamming) weight t. Doing so yields 

m 

^2P[Z j (D) = 1 | Z°(D) = 1] = ^A m (t/m)Q(t/m ;J D) 

< m max {A m (t/m) Q(t/m;D)} 

l<t<m 

< m max {A m (w)Q(w;D)}. 

we[o,i] 

Consequently, we need to control both the LDPC weight enumerator A m (w) and the probability 
Q(w;D) over the range of possible fractional weights w € [0, 1]. 

5.4 Bounding the overlap probability 

The following lemma, proved in Appendix [Dl provides a large deviations bound on the probability 
Q(w;D). 

Lemma 5. For each w £ [0, 1], we have 

-logQ(w;D) < F(5*(w;d c );D)+o(l), (30) 
n 



2 In the latter case, codewords are chosen independently from some ensemble, so that the overlap probability is 
simply equal to f[Z 3 (D) = 1]. Thus, for the simple case of unstructured random coding, the second moment bound 
actually provides the converse to Shannon's rate-distortion theorem for the symmetric Bernoulli source. 
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where for each t E (0, i] and D € (0, i], the error exponent is given by 



F(t; D) := £>log (1 - t)e x * + t +(1-2?) log (1 - t) + te 



X*D. 



(31) 



Here X* : = loe 



— 6+V& 2 — 4ac 
2a 



, where a : = t (1 - 1) (1 - D), b : = (1 - 2D)t 2 , andc:= -t (1 - t) D. 



In general, for any D € (0, the function • ; D) has the following properties. At t = 0, 
it achieves its maximum F(0;D) = 0, and then is strictly decreasing on the interval (0, i], ap- 
proaching its minimum value — [1 — h(D)] as t — > |. Figure [7] illustrates the form of the function 
F(5* (u>; d c ); D) for two different values of distortion D, and for degrees d c G {3,4,5}. Note that 
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Figure 7. Plot of the upper bound (|30|) on the overlap probability — \ogQ(w; D) for different choices 
of the degree d c , and distortion probabilities, (a) Distortion D = 0.1100. (b) Distortion D = 0.3160. 
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increasing d c causes F(5* (lo; d c ); D) to approach its minimum —[1 — h(D)] more rapidly. 

We are now equipped to establish the form of the effective rate-distortion function for any 
compound LDGM/LDPC ensemble. Substituting the alternative form of E[T^] from equation ([2j 
into the second moment lower bound (I25D yields 



-log¥[T n (D) >0] > - 
n n 



logE[r ft (D)] - log, 



l + ^F[Z j (D) = 1 | Z°(D) = 1] 



> R-(l-h(D))- max <| - log A m (w) + - log Q(w; D) 



«)6[o,i] I n 



n 



o{l) 



> R - ( I - h (D)) - n-iax \ R J ^og^H + F ( S*(w;d c ),D)^ - o(l), (32) 



»e[o,i] 
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Minimum achievable rates: (R,D) = (0.50, 0.1100) 
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Figure 8. Plot of the function denning the lower bound (|33|) on the minimum achievable rate for a 
specified distortion. Shown are curves with LDGM top degree d c — 4, comparing the uncoded case 
(no bottom code, dotted curve) to a bottom (4, 6) LDPC code (solid line), (a) Distortion D = 0.1100. 
(b) Distortion D = 0.3160. 
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where the last step follows by applying the upper bound on Q from Lemma El and the relation 
m = Ran = j[^ n - Now letting B(w;d v ,d' c ) be any upper bound on the log of average weight 
enumerator lo s A ^.( w ) ^ we can th en conclude that - logP[T„(.D) > 0] is asymptotically non-negative 
for all rate-distortion pairs (R, D) satisfying 

l-h(D) + F(5*(w;d c ),D) ] 

-, _ B(w;d v ,d' c ) ■ 
1 Rh -I 

Figure [8] illustrates the behavior of the RHS of equation (|33p . whose maximum defines the effective 
rate-distortion function, for the case of LDGM top degree d c = 4. Panels (a) and (b) show the 
cases of distortion D = 0.1100 and D = 0.3160 respectively, for which the respective Shannon rates 
are R = 0.50 and R = 0.10. Each panel shows two plots, one corresponding the case of uncoded 
information bits (a naive LDGM code), and the other to using a rate Rh = 2/3 LDPC code with 
degrees (d v ,d c ) = (4,6). In all cases, the minimum achievable rate for the given distortion is 
obtained by taking the maximum for w G [0,0.5] of the plotted function. For any choices of D, the 
plotted curve is equal to the Shannon bound Rsta, = 1 — h (D) at w = 0, and decreases to for 
w = \. 

Note the dramatic difference between the uncoded and compound constructions (LDPC-coded). 
In particular, for both settings of the distortion (D = 0.1100 and D = 0.3160), the uncoded curves 
rise from their initial values to maxima above the Shannon limit (dotted horizontal line). Con- 



R > max 
tue[o,i] 
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sequently, the minimum required rate using these constructions lies strictly above the Shannon 
optimum. The compound construction curves, in contrast, decrease monotonically from their max- 
imum value, achieved at w = and corresponding to the Shannon optimum. In the following 
section, we provide an analytical proof of the fact that for any distortion D £ [0, |), it is al- 
ways possible to choose finite degrees such that the compound construction achieves the Shannon 
optimum. 

5.5 Finite degrees are sufficient 

In order to complete the proof of Theorem [TJ we need to show that for all rate-distortion pairs 
(R,D) satisfying the Shannon bound, there exist LDPC codes with finite degrees (d v ,d' c ) and a 
suitably large but finite top degree d c such that the compound LDGM/LDPC construction achieves 
the specified (R,D). 

Our proof proceeds as follows. Recall that in moving from equation (|32p to equation (|33p . we 
assumed a bound on the average weight enumerator A m of the form 

— log A,„ H < B(w;d v ,d' c ) + o(l). (34) 
m 

For compactness in notation, we frequently write B(w), where the dependence on the degree pair 
(d v , d' c ) is understood implicitly. In the following paragraph, we specify a set of conditions on this 
bounding function B, and we then show that under these conditions, there exists a finite degree d c 
such that the compound construction achieves specified rate-distortion point. In Appendix [FJ we 
then prove that the weight enumerator of standard regular LDPC codes satisfies the assumptions 
required by our analysis. 

Assumptions on weight enumerator bound We require that our bound B on the weight 
enumerator satisfy the following conditions: 

Al: the function B is symmetric around 5, meaning that B(w) = B{\ — w) for all w 6 [0, 1]. 

A2: the function B is twice differentiable on (0, 1) with B'{\) = and B"{\) < 0. 

A3: the function B achieves its unique optimum at w = ^, where = Rh- 

A4: there exists some e\ > such that B{w) < for all w E (0, ei), meaning that the ensemble 
has linear minimum distance. 
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In order to establish our claim, it suffices to show that for all (R, D) such that R > 1 — h (D), 
there exists a finite choice of d c such that 

max I R + F(5*(w;d c ),D) } < R - [1 - h (D)\ : = A (35) 
we[o,i] I Rh I 

K(w;d c ) 

Restricting to even d c ensures that the function F is symmetric about w = ^\ combined with as- 
sumption A2, this ensures that K is symmetric around |, so that we may restrict the maximization 
to [0, without loss of generality. Our proof consists of the following steps: 

(a) We first prove that there exists an e\ > 0, independent of the choice of d c , such that 
K(w; dc) < A for all 10 G [0, e x ]. 

(b) We then prove that there exists £2 > 0, again independent of the choice of d c , such that 
K(w; d c ) < A for all w € [\ - e 2 , |]. 

(c) Finally, we specify a sufficiently large but finite degree d* that ensures the condition K{w; d*)<A 
for all w G [ei, £2]. 

5.5.1 Step A 

By assumption A4 (linear minimum distance), there exists some e\ > such that B(w) < for all 
w G [0, ei]. Since F(5*(w;d c );D) < for all w, we have K(w;d c ) < < A in this region. Note 
that €1 is independent of d c , since it specified entirely by the properties of the bottom code. 

5.5.2 Step B 

For this step of the proof, we require the following lemma on the properties of the function F: 

Lemma 6. For all choices of even degrees d c > 4, the function G(w;d c ) = F(5* (w;d c ), D) is 
differentiate in a neighborhood of w = h, with 

G(^d c ) = -[l-h(D)}, <-'(i;4) = 0, and G"(±;d c ) = 0. (36) 

See Appendix [E] for a proof of this claim. Next observe that we have the uniform bound 
G(w;d c ) < G(w;A) for all d c > 4 and w G [0, 5]. This follows from the fact that F(u;D) is 
decreasing in u, and that 5*(w; 4) < 6*(w; d c ) for all d c > 4 and w G [0, 5]. Since B is independent 
of d c , this implies that K(w;d c ) < K(w;4) for all w G [0, |]. Hence it suffices to set d c = 4, and 
show that K(w;4) < A for all w G [| — eg, |]. Using Lemma El Assumption A2 concerning the 
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derivatives of B, and Assumption A4 (that B(^) = Rh), we have 

K{\-A) = R-[l-h(D)] = A, 
K'{\-A) = ^^ + G'{\-A) = 0, and 

„a s RB"a) ,.i RB"a) 

K { 2 ]4) = ~ R^T + G ( 2 ;4) = ifo <0 - 

By the continuity of if", the second derivative remains negative in a region around ^, say for all 
w € — €2, tj] for some €2 > 0. Then, for all w G [i — €2, 37], we have for some u; E [w;, ^] the second 
order expansion 

K(w;4) = K{^A)+K'{ 1 --A){w- 1 -) + \k{w-A) (w - ^ 

( l x2 



Thus, we have established that there exists an €2 > 0, independent of the choice of d c , such that 
for all even d c > 4, we have 

K(w;d c ) < K(wA) < A for all w € [\ - e 2 , \\. (37) 

5.5.3 Step C 

Finally, we need to show that K(w;d c ) < A for all w £ [£1,62]. From assumption A3 and the 
continuity of B, there exists some p{e2) > such that 

B(w) < R H [1 - p(e 2 )] for all w < \ - e 2 . (38) 

From Lemma El lim u ^i F(u;D) = F(^;D) = — [1 — /i(D)]. Moreover, as d c — > +00, we have 
<5*(ei; d c ) — > |. Therefore, for any £3 > 0, there exists a finite degree d* c such that 

F(<5*(ei;d*);D) < - [1 - /i (£))] + e 3 . 
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Since F is non-increasing in w, we have F(5*(w;d*);D) < — [1 — h(D)\ + e 3 for all w G [61,62] 
Putting together this bound with the earlier bound (f38|) yields that for all w G [£1,62]: 

^(^4) = + jF ( 5 *( u;;d *) )jD ) 

it// 

< R[l-p(e 2 )]-[l-/ l ( J D)] + e 3 
= {R - [1 -7» (£))]} + (eg -Rp(e 2 )) 
= A + (e 3 -Rp(e 2 )) 

Since we are free to choose e 3 > 0, we may set e 3 = Rp ( £2 ) to yield the claim. 



6 Proof of channel coding optimality 

In this section, we turn to the proof of the previously stated Theorem [2J concerning the channel 
coding optimality of the compound construction. 

If the codeword x £ {0, l} n is transmitted, then the receiver observes V = x W, where W 
is a Ber(p) random vector. Our goal is to bound the probability that maximum likelihood (ML) 
decoding fails where the probability is taken over the randomness in both the channel noise and 
the code construction. To simplify the analysis, we focus on the following sub-optimal (non-ML) 
decoding procedure. Let e n be any non-negative sequence such that e n /n — ► but e^/n — > +00 — say 
for instance, e n = n 2//3 . 

Definition 2 (Decoding Rule:). With the threshold d(n) : = pn + e n , decode to codeword Xi <^=^ 
\\xi © V||i < d(n), and no other codeword is within d{n) of V . 

The extra term e n in the threshold d{n) is chosen for theoretical convenience. Using the following 
two lemmas, we establish that this procedure has arbitrarily small probability of error, whence ML 
decoding (which is at least as good) also has arbitrarily small error probability. 

Lemma 7. Using the suboptimal procedure specified in the definition ([2]), the probability of decoding 
error vanishes asymptotically provided that 

R G B(w) - D(p\\5*(w;d c )*p) < for all w € (0, i], (39) 

where B is any function bounding the average weight enumerator as in equation (|34p . 

Proof. Let N = 2 nR = 2 mRH denote the total number of codewords in the joint LDGM/LDPC 
code. Due to the linearity of the code construction and symmetry of the decoding procedure, we 
may assume without loss of generality that the all zeros codeword n was transmitted (i.e., x = n ). 
In this case, the channel output is simply V = W and so our decoding procedure will fail if and 
only if one the following two conditions holds: 
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(i) either ||W||i > d(n), or 

(ii) there exists a sequence of information bits y G {0, l} m satisfying the parity check equation 
Hy = such that the codeword Gy satisfies \\Gy © W\\\ < d(n). 

Consequently, using the union bound, we can upper bound the probability of error as follows: 

N 

Pew < P[||W||i > d{n)] + ^ P [\\Gy l © W || i < d(n)] . (40) 

i=2 

Since E[||W||i] = pn, we may apply Hoeffdings's inequality [13] to conclude that 

P[||W||i > d(n)} < 2exp(-2^\ ^0 (41) 

by our choice of e n . Now focusing on the second term, let us rewrite it as a sum over the possible 
Hamming weights £ = 1, 2, . . . , m of information sequences (i.e., ||y||i = £) as follows: 

N m 

£>[||CyffiW||i<d(n)] = ^A m (— ) P[||Gj/©W||i > d(n) | ||y||i =£], 

where we have used the fact that the (average) number of information sequences with fractional 
weight I I'm is given by the LDPC weight enumerator A m (^r). Analyzing the probability terms 
in this sum, we note Lemma [9] (see Appendix [Al) guarantees that Gy has i.i.d. Ber(5*(— ; d c )) 
elements, where S*( ■ ; d c ) was defined in equation ([29]) . Consequently, the vector Gy © W has i.i.d. 
Ber(<5(^) * p) elements. Applying Sanov's theorem [11] for the special case of binomial variables 
yields that for any information bit sequence y with £ ones, we have 

P[||GyffiW||i >d(n) | =£) < /(n)2~ nD ( p|l<5( ^H, (42) 

for some polynomial term f(n). We can then upper bound the second term in the error bound (|4Up 
as 

VpniGV© WIN < din)} < f(m) expl max \mB{—) + o(m) - nD (p\\S(—) *p) 
L J I i<e<m I m \ m J 

where we have used equation (I42h . as well as the assumed upper bound (|34h on A m in terms of B. 
Simplifying further, we take logarithms and rescale by m to assess the exponential rate of decay, 
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thereby obtaining 



1 r f 1 / f \ i 

— log VP[||(V Wlh < d(n)] < max B(— ) D[p\\6(—)*p) +o(l) 

m f^ 1 L J i<^<m L to R G \ to J . 



< max 
WE[0,1] 



B(w)--±-D(p\\5(w)*p) --oil). 
Rg 



and establishing the claim. □ 

Lemma 8. For any p G (0,1) and total rate R : = Rg Rh < 1 — h (p), there exist finite choices 
of the degree triplet (d c ,d v ,d' c ) such that (|39p is satisfied. 

Proof. For notational convenience, we define 

L{w) := R G B(w)-D(p\\6*(w;d c )*p). (43) 

First of all, it is known [T7j that a regular LDPC code with rate Rh = % < 1 and d v > 3 has linear 
minimum distance. More specifically, there exists a threshold v* = v*(d v ,d c ) such that B(w) < 
for all w G [0,z/*]. Hence, since B(w) - D (p\\5*(w; d c ) * p) > for all w £ (0,1), for w G (0,z/*], we 
have L(w) < 0. 

Turning now to the interval [v*, |], consider the function 

L(w) := Rh(w) - D{p\\5*(w;d c )*p). (44) 

Since i?(w) < Rnh (w), we have L(w) < L(w), so that it suffices to upper bound L. Observe that 
L(i) = R — (1 — /i(p)) < by assumption. Therefore, it suffices to show that, by appropriate 
choice of d c , we can ensure that L(w) < L(^). Noting that L is infinitely differentiable, calculating 
derivatives yields L'(^) = and L"(^) < 0. (See Appendix [G] for details of these derivative 
calculations.) Hence, by second order Taylor series expansion around w = |, we obtain 

L(w) = L{ 1 -) + \l"(w)(w- 1 -)\ 

where id G [w, 5]. By continuity of L" , we have L"(w) < for all in some neighborhood of ^, 
so that the Taylor series expansion implies that L(w) < L(^) for all w in some neighborhood, say 

It remains to bound L on the interval [z/*,/x]. On this interval, we have L(w) < Rh(fi) — 
D (p\\6*(u *; d c ) * p). By examining equation (f29l) from Lemma EJ we see that by choosing d c 
sufficiently large, we can make 5*(u*;d c ) arbitrarily close to |, and hence D (p\\5* (v* ; d c ) * p) 
arbitrarily close to 1 — h (p) . More precisely, let us choose d c large enough to guarantee that 
D (p\\5* (v* ; d c ) * p) < (1 — e) (1 — h(p)), where e = R ^_^^ ■ With this choice, we have, for all 
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w € [i/*, fj] , the sequence of inequalities 



L(w) 



< Rh 

< Rh (n) 
= R-(l 



D(p\\5*(u*;d c )*p) 
[(l-h(p))-R(l-h^))} 
h(p)) < 0, 



which completes the proof. 



□ 



7 Discussion 

In this paper, we established that it is possible to achieve both the rate-distortion bound for 
symmetric Bernoulli sources and the channel capacity for the binary symmetric channel using 
codes with bounded graphical complexity. More specifically, we have established that there exist 
low-density generator matrix (LDGM) codes and low-density parity check (LDPC) codes with 
finite degrees that, when suitably compounded to form a new code, are optimal for both source 
and channel coding. To the best of our knowledge, this is the first demonstration of classes of codes 
with bounded graphical complexity that are optimal as source and channel codes simultaneously. 
We also demonstrated that this compound construction has a naturally nested structure that can 
be exploited to achieve the Wyner-Ziv bound [UJ for lossy compression of binary data with side 
information, as well as the Gelfand-Pinsker bound [19] for channel coding with side information. 

Since the analysis of this paper assumed optimal decoding and encoding, the natural next step 
is the development and analysis of computationally efficient algorithms for encoding and decoding. 
Encouragingly, the bounded graphical complexity of our proposed codes ensures that they will, with 
high probability, have high girth and good expansion, thus rendering them well-suited to message- 
passing and other efficient decoding procedures. For pure channel coding, previous work [TBI ESI SI] 
has analyzed the performance of belief propagation when applied to various types of compound 
codes, similar to those analyzed in this paper. On the other hand, for pure lossy source coding, our 
own past work [Sj provides empirical demonstration of the feasibility of modified message-passing 
schemes for decoding of standard LDGM codes. It remains to extend both these techniques and their 
analysis to more general joint source/channel coding problems, and the compound constructions 
analyzed in this paper. 
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A Basic property of LDGM codes 



For a given weight w £ (0, 1), suppose that we enforce that the information sequence y £ {0, l} m 
has exactly \wm\ ones. Conditioned on this event, we can then consider the set of all codewords 
X(w) £ {0, l} n , where we randomize over low-density generator matrices G chosen as in step (a) 
above. Note for any fixed code, X(w) is simply some codeword, but becomes a random variable 
when we imagine choosing the generator matrix G randomly. The following lemma characterizes 
this distribution as a function of the weight w and the LDGM top degree d c : 

Lemma 9. Given a binary vector y £ {0, l} m with a fraction w of ones, the distribution of 
the random LDGM codeword X{w) induced by y is i.i.d. Bernoulli with parameter 5*(w;d c ) = 



1 - (1 - 2w) 



d c 



Proof. Given a fixed sequence y £ {0, l} m with a fraction w ones, the random codeword bit Xi(w) 
at bit i is formed by connecting d c edges to the set of information bits|f| Each edge acts as an i.i.d. 
Bernoulli variable with parameter w, so that we can write 

X t (w) = Vi®V 2 ®...®V dc , (45) 

where each ~ Ber(u;) is independent and identically distributed. A straightforward calculation 
using z-transforms (see [T7j) or Fourier transforms over GF(2) yields that Xi(w) is Bernoulli with 
parameter 5*(w;d c ) as defined. □ 

B Bounds on binomial coefficients 

The following bounds on binomial coefficients are standard (see Chap. 12, |llj): 

, fk\ log(n + 1) 1 , AA , fk\ , t . 

h - - "<-log , <M- ■ 46 

\n J n n \k J \n J 

Here, for a £ (0,1), the quantity h(a) := —a log a — (1 — a)log(l — a) is the binomial entropy 
function. 



3 In principle, our procedure allows two different edges to choose the same information bit, but the probability of 
such double-edges is asymptotically negligible. 
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C Proof of Lemma [4] 

First, by the definition of T n {D), we have 



E[T n 2 (D)] = E 



N-l JV-1 

2 ^ Z\D)Z^D) 

i=l j=0 
N-l 

= E[T n ] + = 1» Z\D) = 1]. 

i=0 j^i 

To simplify the second term on the RHS, we first note that for any i.i.d Bernoulli(^) sequence 
S G {0,1}™ and any codeword X 7 , the binary sequence S' := 5 © X 3 is also i.i.d. Bernoulli(^). 
Consequently, for each pair i ^ j, we have 



[Z\D) = 1, Zi{D) = 1] 



|^ 5||i < Dn, \\X j © 5||i < Dn] 
\X { © 5' ||i < Dn, \\X j © 5' ||i < Dn] 
\X* © X © 5||i < Dn, ||5||i < Dn) . 



Note that for each j ^ i, the vector X 1 © X 3 is a non-zero codeword. For each fixed i, summing 
over j y£ i can be recast as summing over all non-zero codewords, so that 



Y J W[Z i {D) = l,&{D) = l} 

i+3 



N-l 



^^PfllXffiX'ffiSHi <Dn,||5||i <Dn] 
i=0 j^i 
N-l 



i=0 k^O 



J^P [\\X k © 5||i < Dn, \\S\\i < Dn 
[||X fe © 5||i < Dn, \\S\\i < Dn 

k^O 

2 nR F [Z°{D) = 1] J2 ¥ [ zk ( D ) = 1 I Z°(D) = 1 
E[T n ] ^p[z fc (L>) = l | Z°(D) 



k^O 



thus establishing the claim. 
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D Proof of Lemma [5] 



We reformulate the probability Q(w, D) as follows. Recall that Q involves conditioning the source 
sequence S on the event H^Hi < Dn. Accordingly, we define a discrete variable T with distribution 



P(T = t) 



EDn fn\ 
s=0 \s) 



for t = 0, 1, . . . , Dn, 



representing the (random) number of Is in the source sequence S. Let U% and Vj denote Bernoulli 
random variables with parameters 1 — 5*(w;d c ) and S*(uu;d c ) respectively. With this set-up, con- 
ditioned on codeword j having a fraction wn ones, the quantity Q(w, D) is equivalent to the 
probability that the random variable 



W 



EL^+E-=r^ ifT>i 

if T = 



is less than Dn. To bound this probability, we use a Chernoff bound in the form 

-logPpy <Dn] < inf ( -logMiv(A)- AD ) . 
n a<o V n I 



(47) 



(48) 



We begin by computing the moment generating function Mvp. Taking conditional expectations and 
using independence, we have 



Dn 



M w {\) = ^P[T = t] [M[/(A)]* [Mv(A)] 



n—t 



t=0 



Here the cumulant generating functions have the form 



logMbj(A) 
log M v (A) 



log 
log 



[l-5)e x + 5 
[1-S) + 5e x 



and 



(49a) 
(49b) 



where we have used (and will continue to use) 5 as a shorthand for 5*(w; d c ). 

Of interest to us is the exponential behavior of this expression in n. Using the standard entropy 
approximations to the binomial coefficient (see Appendix [B]), we can bound M^(A) as 



/(n) £ew[n{h(J^-h(D) + ~]ogMu(\)+(l-^ logM v (A)}], 



(50) 



ait) 
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where f(n) denotes a generic polynomial factor. Further analyzing this sum, we have 



1 , / \ 1 / n log fin) log(nD) 

-log^sW < - max log 5 (t) + ^^ + ^ ^ 

n n o<t<Dn n n 

= max j/J 1 J -h{D) + -\ogMu(\)+ ( 1- - J logMy(A)) + o(l) 

0<t<Dn [V n / n V n / J 

< max {h(u) - h(D) +u log R%(A) + (1 - it) log My (A)} + o(l). 
«e[o,D] 

Combining this upper bound on - logMjy(A) with the Chernoff bound (|48p yields that 

-logP[W<I>nl < inf max G(u, A; 5) + o(l) (51) 

n x<Oue[o,D] 

where the function G takes the form 

G(u,X;6) := h(u)-h(D)+ulogM u (X) + (l-u)logM v (X)-XD. (52) 

Finally, we establish that the solution (u*,X*) to the min-max saddle point problem (|51|) is 
unique, and specified by u* = D and A* as in Lemma[5j First of all, observe that for any 5 € (0, 1), 
the function G is continuous, strictly concave in u and strictly convex in A. (The strict concavity 
follows since h (u) is strictly concave with the remaining terms linear; the strict convexity follows 
since cumulant generating functions are strictly convex.) Therefore, for any fixed A < 0, the 
maximum over u S [0, D] is always achieved. On the other hand, for any D > 0, u S [0, D\ and 
5 € (0, 1), we have G(u; \;t) — > +oo as A — > — oo, so that the infimum is either achieved at some 
A* < 0, or at A* = 0. We show below that it is always achieved at an interior point A* < 0. Thus 
far, using standard saddle point theory [21], we have established the existence and uniqueness of 
the saddle point solution (u*,A*). 

To verify the fixed point conditions, we compute partial derivatives in order to find the optimum. 
First, considering u, we compute 

BC 1 — ii 

— (u,X;5) = log +lo g M t/ (A)-logM v (A) 

ou u 



1-11 



- log h log 

u 

Solving the equation ^j("U, A; 5) = yields 



l-5)e x + 5 -log (1-S) + Se 



u> = ^!^D + —±—n-D)>0. (53) 
1 + exp(A) 1 + exp(A) 

Since D < ^, a bit of algebra shows that u' > D for all choices of A. Since the maximization is 
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constrained to [0, D], the optimum is always attained at u* = D. 

Turning now to the minimization over A, we compute the partial derivative to find 



5 exp(A) 



D. 



[l-6)exp(X)+5 v '(1 -5) +5exp(A) 
Setting this partial derivative to zero yields a quadratic equation in exp(A) with coefficients 



a = 5 (1 - 5) (1 - D) 

b = u(l - 5) 2 + (1 - u)8 2 - D [8 2 + (1 - 8) 2 ] . 
c = -D6(l-S). 



The unique positive root p* of this quadratic equation is given by 



p*(5,D,u) 



2a L 



-b + y/b 2 - Aac 



(54a) 
(54b) 
(54c) 



(55) 



It remains to show that p* < 1, so that A* : = log p* < 0. A bit of algebra (using the fact a > 0) 
shows that p* < 1 if and only if a + b + c > 0. We then note that at the optimal u* = D, we have 
b=(l-2D)5 2 , whence 

a + b + c = 5{\-5){l-D) + (l-2D)5 2 - DS(l - 6) 
= (I -2D) 5 > 

since D < h and 5 > 0. Hence, the optimal solution is A* : = log p* < 0, as specified in the lemma 
statement. 



E Proof of Lemma [6] 

A straightforward calculation yields that 

G(\) = F(S*(^ de); D) = F(±; D) = - (1 - h (D)) 

as claimed. Turning next to the derivatives, we note that by inspection, the solution X*(t) defined in 
Lemma [5] is twice continuously differentiable as a function of t. Consequently, the function F{t, D) 
is twice continuously differentiable in t. Moreover, the function 5*(w;d c ) is twice continuously 
differentiable in w. Overall, we conclude that G(w) = F(d*(w;d c );D) is twice continuously differ- 
entiable in w, and that we can obtain derivatives via chain rule. Computing the first derivative, 
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we have 

G'(\) = 5>(\)F'{5*( l -;d c );D) = 

since 6'(w) = —d c (1 — 2«;) o!c ~ 1 , which reduces to zero at w = \. Turning to the second derivative, 
we have 

G"{\) = S"{\)F'{5*{\ ] d c ) ] D) + (5'{\)j F"(5*(±;d c y,D) = S" \\) F' '(<T : (~; d c ); D) . 

We again compute 8"{w) = 2d c (d c — 1)(1 — 2w) dc ~ 2 , which again reduces to zero at w = \ since 
d c > 4 by assumption. 

F Regular LDPC codes are sufficient 

Consider a regular (d v , d' c ) code from the standard Gallager LDPC ensemble. In order to complete 
the proof of Theorem CO we need to show for suitable choices of degree (d v ,d' c ), the average weight 
enumerator of these codes can be suitably bounded, as in equation (fM|) . by a function B that 
satisfies the conditions specified in Section [531 

It can be shown |17[ I22j that for even degrees d' c , the average weight enumerator of the regular 
Gallager ensemble, for any block length m, satisfies the bound 

— logA m (u>) = B(w;d v ,d') + o(l). 
m 

The function B in this relation is defined for w G [0, ^] as 

B(w; d v ,d' c ) := (1 - d v )h H - (1 - R H ) + d v inf | j log ((1 + e A )< + (1 - e x f c ) ~ w\\ , (56) 

and by B(w) = B(w — ^) for w G Given that the minimization problem (|56p is strictly 

convex, a straightforward calculation of the derivative shows the optimum is achieved at A* , where 
A* < is the unique solution of the equation 

. (i + eY^-q-eY-- 1 _ w (57) 

Some numerical computation for Rh = 0.5 and different choices (d v ,d' c ) yields the curves shown 
in Fig. M 

We now show that for suitable choices of degree (d v ,d' c ), the function B defined in equation ([56]) 
satisfies the four assumptions specified in Section 15.51 First, for even degrees d' c , the function B 
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LDPC weight enumerators 




Weight 

Figure 9. Plots of LDPC weight enumerators for codes of rate Rr = 0.5, and check degrees 
d' c e {6,8,10}. 

is symmetric about w = ^, so that assumption (Al) holds. Secondly, we have B(w) < Rh, and 
moreover, for w = ^, the optimal A*(|) = 0, so that B(^) = Rh, and assumption (A3) is satisfied. 
Next, it is known from the work of Gallager [17] . and moreover is clear from the plots in Fig. [9l 
that LDPC codes with d v > 2 have linear minimum distance, so that assumption (A4) holds. 

The final condition to verify is assumption (A2), concerning the differentiability of B. We 
summarize this claim in the following: 

Lemma 10. The function B is twice continuously differentiable on (0, 1), and in particular we 
have 

= 0, and < 0. (58) 

Proof. Note that for each fixed w G (0, 1), the function 

/(A) = J 7 log((l + e A )< + (l-e A )<) = ^log((e- A + l)< + (e- A -l)<)+A 

is strictly convex and twice continuously differentiable as a function of A. Moreover, the function 
f*(w) : = inf^<o {/(A) — Xw} corresponds to the conjugate dual [21J of /(A) + I<o(A). Since the 
optimum is uniquely attained for each w € (0, 1), an application of Danskin's theorem |4| yields 
that /* is differentiable with ^f*(w) = —A* (to), where A* is defined by equation (|57|) . Putting 
together the pieces, we have B'(w) = (1 — d v )h'{w) — d v \*{w). Evaluating at w = ^ yields 
B'{\) = - 4A*(0) = as claimed. 

We now claim that X*(w) is differentiable. Indeed, let us write the defining relation (|57p for 
X*(w) as F(X, w) = where -F(A, w) : = /'(A) — w. Note that F is twice continuously differentiable 
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in both A and w; moreover, exists for all A < and w, and satisfies §r(A, w) = /"(A) > 
by the strict convexity of /. Hence, applying the implicit function theorem [1] yields that \*{w) 
is differentiable, and moreover that j^(w) = l/f"(\*(w)). Hence, combined with our earlier 



calculation of B' , we conclude that B"(w) 



[1 - d v )h"(w) 



d, 



-. Our final step is to 



" /"(A(io)) ' 

compute the second derivative /". In order to do so, it is convenient to define g = log/', and 
exploit the relation g'f = f". By definition, we have 



9(A) 



A + log 



(l + e A ) 



X\d'-l 



(i-O 



\\d'-l 



log 



(l + e y c + (i_ e 



\\d' 



whence 



, w (i + e y- 2 + (i 



„A\<f'-2 



l + e\d' c -l) 



(l +e A)d' c -l_( 1 _ e A)^-l 



eV (1 + e 



\\d'-l 



,A\d'-l 



(l + e A ) d c + (l- e A)^ 



Evaluating at to = ^ corresponds to A(0) = 0, so that 



r(A(-)) 



/W(0) 



i + K- 1 )^!-^- 



d'-l 



2 d 'c 



Consequently, combining all of the pieces, we have 



B''{w) = {l-d v )h!'{\)-d v j^y ) 



d v - 1 



Ad v < 



as claimed. 



□ 



G Derivatives of L 

Here we calculate the first and second derivatives of the function L defined in equation (|44p . The 
first derivative takes the form 

jit \ pi 1 ~ V , ^'CMc) n s *'(«;dc) 

L (v) = /tlog ^Ptt — ~T~\ — (1 — p)- 



S(v;d c ) l-5(v;d c ) 
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where 5'(v;d c ) = d c (l — 2v) dc 1 . Since d'(h;d c ) = 0, we have L'{\) = as claimed. Second, using 
chain rule, we calculate 

L"(v) = R\ 1 + 1] | S "( v -> d cMv,d c ) - [5'{v,d c )f 

L l — V IT 



[5(v;d c )]< 



5"(v;d c )[l-5(v;d c )]+[5'(v;d c )}' 
[ P) [l-5(v;d c W 



and 5"{v;d c ) = -d c (d c - 1) (1 - 2v) dc ~ 2 . Now for d c > 2, we have 8"(\) = 0, so that L"(\) 
—AR < as claimed. 
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